Bank Marketing Data Set¶
Goal¶
Using the data collected from existing customers, build a model that will help the marketing team identify potential customers who are relatively more likely to subscribe term deposit and thus increase their hit ratio.
Attribute Information:¶
Bank client data:¶
- age (numeric)
- job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')
- marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)
- education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')
- default: has credit in default? (categorical: 'no','yes','unknown')
- balance: balance in account
- housing: has housing loan? (categorical: 'no','yes','unknown')
- loan: has personal loan? (categorical: 'no','yes','unknown')
Related with the last contact of the current campaign:¶
- contact: contact communication type (categorical: 'cellular','telephone')
- month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')
- day: last contact day of the month
- duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.
Other attributes:¶
- campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
- pdays: number of days that passed by after the client was last contacted from a previous campaign (-1 tells us the person has not been contacted or contact period is beyond 900 days)
- previous: number of contacts performed before this campaign and for this client (numeric)
- poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')
Output variable (desired target):¶
- Target: Tell us has the client subscribed a term deposit. (Yes, No)
Deliverable – 1 (Exploratory data quality report reflecting the following) – (20)¶
Univariate analysis (12 marks)
- Univariate analysis – data types and description of the independent attributes which should include (name, meaning, range of values observed, central values (mean and median), standard deviation and quartiles, analysis of the body of distributions / tails, missing values, outliers.
- Strategies to address the different data challenges such as data pollution, outlier’s treatment and missing values treatment.
- Please provide comments in jupyter notebook regarding the steps you take and insights drawn from the plots.
Multivariate analysis (8 marks)
- Bi-variate analysis between the predictor variables and target column. Comment on your findings in terms of their relationship and degree of relation if any. Visualize the analysis using boxplots and pair plots, histograms or density curves. Select the most appropriate attributes.
- Please provide comments in jupyter notebook regarding the steps you take and insights drawn from the plots
Deliverable – 2 (Prepare the data for analytics) – (10)¶
- Ensure the attribute types are correct. If not, take appropriate actions.
- Get the data model ready.
- Transform the data i.e. scale / normalize if required
- Create the training set and test set in ratio of 70:30
Deliverable – 3 (create the ensemble model) – (30)¶
- First create models using Logistic Regression and Decision Tree algorithm. Note the model performance by using different matrices. Use confusion matrix to evaluate class level metrics i.e. Precision/Recall. Also reflect the accuracy and F1 score of the model. (10 marks)
- Build the ensemble models (Bagging and Boosting) and note the model performance by using different matrices. Use same metrics as in above model. (at least 3 algorithms) (15 marks)
- Make a DataFrame to compare models and their metrics. Give conclusion regarding the best algorithm and your reason behind it. (5 marks)
In [3]:
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set_style(style='darkgrid')
import plotly.express as px
from sklearn.model_selection import train_test_split as tts
from sklearn import tree
from sklearn import metrics as mtr
In [4]:
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import cross_validate
from yellowbrick.classifier import ROCAUC
In [5]:
pd.options.display.float_format = '{:,.4f}'.format
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:80% !important; }</style>"))
In [6]:
df = pd.read_csv('bank-full.csv')
In [7]:
df.head()
Out[7]:
In [8]:
df.rename(columns={'Target':'target'}, inplace=True)
In [9]:
df.shape
Out[9]:
In [10]:
df[df.duration == 0]
Out[10]:
In [11]:
# Dropping observations where the duration is 'zero' so as to not influence the model
df.drop(df[df.duration == 0].index, axis=0, inplace=True)
dfM = df.copy()
Deliverable1 - EDA¶
Univariate analysis (12 marks)¶
- Univariate analysis – data types and description of the independent attributes which should include (name, meaning, range of values observed, central values (mean and median), standard deviation and quartiles, analysis of the body of distributions / tails, missing values, outliers.
- Strategies to address the different data challenges such as data pollution, outlier’s treatment and missing values treatment.
- Please provide comments in jupyter notebook regarding the steps you take and insights drawn from the plots.
Number of observations, the data-types and if they are null/not-null info¶
In [12]:
df.info()
Explicit checking for NULLs¶
In [13]:
df.isnull().sum()
Out[13]:
Inference: No 'NULL' values in the data-set, hence no corrections are necessary¶
Checking for 'zero' values to determine missing data or if the 'zeros' are part of the data¶
In [14]:
df[df==0].count()
Out[14]:
Inference: Zero values in 'balance' feature and in 'previous' feature are valid data hence no corrections are necessary¶
Five-number summary for the dataset¶
In [15]:
df.describe(include='all').T
Out[15]:
Inference: There is heavy skewness (right) with the following features: balance, duration, campaign, pdays and previous¶
Summary view of distribution of all continous variables in the data-set¶
In [16]:
df.hist(bins=30, figsize=[30,20])
Out[16]:
Inference:¶
- Day is evenly distributed (as expected)
- Age is close to normal distribution (as expected)
- Rest of the continuous variables (balance, campaign, pdays, previous & duration) are heavily skewed right.
Data-set management strategy:¶
- No NULLs or Zeros in the data-set which is a good thing; hence no cleansing necessary.
- The issue to be tackled is the heavy data skewness and the strategy that I'm employing is to convert these heavily skewed continuous variables into categorical variables and have them 1-hot-encoded
- Analysing individual features will help create the appropriate buckets for this continuous to categorical variable conversion
2. Multivariate analysis (8 marks)¶
- Bi-variate analysis between the predictor variables and target column. Comment on your findings in terms of their relationship and degree of relation if any. Visualize the analysis using boxplots and pair plots, histograms or density curves. Select the most appropriate attributes.
- Please provide comments in jupyter notebook regarding the steps you take and insights drawn from the plots
Comparing features against one another for the two possible outcomes (target)¶
In [17]:
plt.figure(figsize=[30,20])
sns.set(font_scale=1.15)
sns.pairplot(dfM, hue='target')
plt.show()
Observations:¶
- There is a lot more campaign effort for consumers at the lower age spectrum compared to older consumers, however there are equally higher number of non-conversions (target = no)
- Age profile of consumers is similar between the two outcomes
- Day of month distribution is multi-modal - understandable based on the weekends interpersing through the month
- Consumer with target=yes have the characteristic trait of showing highest spike in lower end of the spectrum (which also includes -1 potentially indicating no contact)
- Similar (to #4) observation with the feature 'previous' showing highest spike in lower end of the spectrum (no previous contact)
Analysing the correlation between the features¶
In [18]:
corr = dfM.corr()
plt.figure(figsize=[20,15])
sns.set(font_scale=1.15)
sns.heatmap(corr,annot=True,vmin=-1,vmax=1,center=0, linewidth=0.2, fmt='.2f', cmap='coolwarm')
plt.show()
Inference¶
- Besides the strong positive correlation between pdays and previous (as described earlier), there are no significant correlation between these features
'duration' feature analysis¶
In [19]:
fig = px.histogram(df, x='duration', marginal='box', color='target')
fig.show()
Observation : Median duration seems to be longer in conversion (target = yes)¶
In [20]:
# Converting duration from continuous variable to categorical variable by grouping them into buckets
durationbins=[0,50,77,100,125,150,180,215,260,320,410,600,5000]
durationlabels=['1-50','50-77','77-100','100-125','125-150','150-180','180-215','215-260','260-320','320-410','410-600','600+']
df['durationCat'] = pd.cut(df.duration, bins=durationbins,labels=durationlabels)
df.durationCat.value_counts()
Out[20]:
'previous' feature analysis¶
In [21]:
fig = px.histogram(df[df.previous < 50], x='previous', marginal='box', color='target')
fig.show()
Observations :¶
- Ignoring the outliers, the 'previous' feature seem to be having the same kind of distribution of observations/data across both outcome groups.
- From the violin chart below, there seems to be higher ratio of 'Zero contacts' in the conversion group (target = yes)
In [22]:
fig = px.violin(df, color='target', y='previous', width=800)
fig.show()
In [23]:
df[df.previous == 0].target.value_counts()
Out[23]:
In [24]:
df[df.previous > 0].target.value_counts()
Out[24]:
In [25]:
# Converting previous from continuous variable to categorical variable by grouping them into buckets
previousbins=[-2,0,1,2,4,300]
previouslabels=['0','1','2','3-4','5+']
df['previousCat'] = pd.cut(df.previous, bins=previousbins,labels=previouslabels)
df.previousCat.value_counts().sort_values()
Out[25]:
'pdays' feature analysis¶
In [26]:
fig = px.histogram(df, x='pdays', marginal='box', color='target')
fig.show()
In [27]:
df[df.pdays == -1].target.value_counts()
Out[27]:
In [28]:
df[df.pdays > -1].target.value_counts()
Out[28]:
Comparing the ratio of 'pdays' for target=yes/no and 'previous' for target = yes/no, we can conclude these two are positively correlated. We can validate that with linear-scatter plot below¶
In [29]:
fig = px.scatter(df, x='previous', y='pdays', trendline="ols", width=600)
fig.show()
Given 'previous' and 'pdays' have a very strong linear correlation, we can drop one of these features - dropping 'pdays'¶
In [30]:
df.drop('pdays',axis=1, inplace=True)
'campaign' feature analysis¶
In [31]:
fig = px.histogram(df, x='campaign', marginal='box', color='target')
fig.show()
'Campaign' seems to have very similar distribution between the two outcome groups (target = yes/no)¶
In [32]:
# Converting campaign from continuous variable to categorical variable by grouping them into buckets
campaignbins=[-1,1,2,3,5,70]
campaignlabels=['1','2','3','4-5','5+']
df['campaignCat'] = pd.cut(df.campaign, bins=campaignbins,labels=campaignlabels)
df.campaignCat.value_counts().sort_values()
Out[32]:
'balance' feature analysis¶
In [33]:
fig = px.histogram(df, x='balance', marginal='box', color='target')
fig.show()
Having a negative balance in the account hasn't deterred from conversion, although the median for the data pertaining to positive conversion seems to be higher than the other group¶
In [34]:
# Converting balance from continuous variable to categorical variable by grouping them into buckets
balancebins=[-9000,-50,0,60,150,250,400,600,900,1400,2200,4000,103000]
balancelabels=['<(50)','(50)-0','0-60','60-150','150-250','250-400','400-600','600-900','900-1.4K','1.4K-2.2K','2.2K-4K','4K+']
df['balanceCat'] = pd.cut(df.balance, bins=balancebins,labels=balancelabels)
In [35]:
df.balanceCat.value_counts().sort_values()
Out[35]:
'age' feature analysis¶
In [36]:
fig = px.histogram(df, x='age', marginal='box', color='target')
fig.show()
Similar pattern of data between the two outcome groups.¶
'day' feature analysis¶
In [37]:
fig = px.histogram(df, x='day', marginal='box', color='target')
fig.show()
Similar pattern of data between the two outcome groups.¶
Deliverable – 2 (Prepare the data for analytics) – (10)¶
- Ensure the attribute types are correct. If not, take appropriate actions.
- Get the data model ready.
- Transform the data i.e. scale / normalize if required
- Create the training set and test set in ratio of 70:30
In [38]:
df.info()
In [39]:
# With the addition of categorical variables, the original features are removed
df.drop(['duration','previous','campaign','balance'],axis=1, inplace=True)
In [40]:
# Categorical variables are converted from object datatype to category datatype
for col in ['job','marital','education','default','housing','loan','contact','month','poutcome']:
df[col] = df[col].astype('category')
In [41]:
df.info()
In [48]:
# Splitting the dependent variable from features
dfy = df.target.copy()
dfX = df.drop('target', axis=1)
In [49]:
# 1-hot-encoding the features / independent variables
dfX = pd.get_dummies(dfX)
In [50]:
dfX.info()
In [51]:
# Convert dependent variable (outcome) to binary (0 & 1)
ymap = {'yes':1, 'no':0}
dfy = dfy.replace(ymap)
dfy.value_counts()
Out[51]:
In [52]:
# Splitting the data for training
dfX_tr, dfX_ts, dfy_tr, dfy_ts = tts(dfX, dfy, test_size=0.3, random_state=6, stratify=dfy)
In [53]:
dfy_tr.value_counts(normalize=True)*100
Out[53]:
In [54]:
dfy_ts.value_counts(normalize=True)*100
Out[54]:
Managed to get equal distribution of outcome values in both training & testing split¶
Deliverable – 3 (create the ensemble model) – (30)¶
- First create models using Logistic Regression and Decision Tree algorithm. Note the model performance by using different matrices. Use confusion matrix to evaluate class level metrics i.e. Precision/Recall. Also reflect the accuracy and F1 score of the model. (10 marks)
- Build the ensemble models (Bagging and Boosting) and note the model performance by using different matrices. Use same metrics as in above model. (at least 3 algorithms) (15 marks)
- Make a DataFrame to compare models and their metrics. Give conclusion regarding the best algorithm and your reason behind it. (5 marks)
3a) Logistic Regression¶
In [55]:
from sklearn.linear_model import LogisticRegression
logit = LogisticRegression(solver='newton-cg')
In [56]:
logit.fit(dfX_tr,dfy_tr)
print('Training score : ',logit.score(dfX_tr, dfy_tr))
logit.fit(dfX_ts,dfy_ts)
print('Test score : ',logit.score(dfX_ts, dfy_ts))
In [57]:
y_pred1 = logit.predict(dfX_ts)
cmat1 = mtr.confusion_matrix(dfy_ts,y_pred1, labels=[1,0])
dfcm1 = pd.DataFrame(cmat1, index=['1','0'],columns=['1','0'])
plt.figure(figsize=[5,3])
sns.set(font_scale=1.2)
fig = sns.heatmap(dfcm1, annot=True, fmt='d',linewidth=0.5, cbar=False)
plt.tick_params(axis='both', which='major', labelbottom = False, bottom=False, top = False, labeltop=True)
plt.ylabel('Predicted\n')
plt.show()
In [58]:
roc1 = ROCAUC(logit)
roc1.fit(dfX_tr, dfy_tr)
roc1.score(dfX_ts, dfy_ts)
Out[58]:
In [59]:
# print classification report
print(mtr.classification_report(dfy_ts,y_pred1))
rocauc1 = roc1.score(dfX_ts, dfy_ts)
prec1 = mtr.precision_score(dfy_ts, y_pred1, average='macro')
recl1 = mtr.recall_score(dfy_ts, y_pred1, average='macro')
f1scr1 = mtr.f1_score(dfy_ts, y_pred1, average='macro')
accr1 = mtr.accuracy_score(dfy_ts, y_pred1)
print('ROC-AUC (by scoring) : ',rocauc1)
print('Precision Macro : ',prec1)
print('Recall Macro : ',recl1)
print('F1 Macro : ',f1scr1)
print('Accuracy : ',accr1)
compare = pd.DataFrame('',index=['Precision Macro','Recall Macro','F1 Macro','Accuracy','ROC-AUC'],columns=['Logit','DecTree','Bagging','AdaBoost','GradBoost','RandomForest'])
compare.Logit = np.array([prec1,recl1,f1scr1,accr1,rocauc1])
3b) Decision Tree¶
In [60]:
from sklearn.tree import DecisionTreeClassifier as dtc
In [56]:
param_grid = {
'max_depth':[10,15,20,25,50]
}
grid = GridSearchCV(dtc(criterion='gini',splitter='best'),param_grid,refit=True, verbose=True, n_jobs=-1, scoring='recall_macro')
gfit = grid.fit(dfX_tr,dfy_tr)
# print best parameter after tuning
print('Best Params : ',grid.best_params_)
# print classification report
grid_predictions = grid.predict(dfX_ts)
print(mtr.classification_report(dfy_ts, grid_predictions))
print("Best score: %0.3f" % gfit.best_score_)
In [61]:
dtree = dtc(criterion='gini',splitter='best',max_depth=20 )
dtree.fit(dfX_tr, dfy_tr)
print('Training Score : ',dtree.score(dfX_tr, dfy_tr))
print('Test Score : ',dtree.score(dfX_ts, dfy_ts))
y_pred2 = dtree.predict(dfX_ts)
cmat2 = mtr.confusion_matrix(dfy_ts,y_pred2, labels=[1,0])
dfcm2 = pd.DataFrame(cmat2, index=['1','0'],columns=['1','0'])
plt.figure(figsize=[5,3])
sns.set(font_scale=1.2)
fig = sns.heatmap(dfcm2, annot=True, fmt='d',linewidth=0.5, cbar=False)
plt.tick_params(axis='both', which='major', labelbottom = False, bottom=False, top = False, labeltop=True)
plt.ylabel('Predicted\n')
plt.show()
In [62]:
roc2 = ROCAUC(dtree)
roc2.fit(dfX_tr, dfy_tr)
roc2.score(dfX_ts, dfy_ts)
Out[62]:
In [63]:
# print classification report
print(mtr.classification_report(dfy_ts,y_pred2))
rocauc2 = roc2.score(dfX_ts, dfy_ts)
prec2 = mtr.precision_score(dfy_ts, y_pred2, average='macro')
recl2 = mtr.recall_score(dfy_ts, y_pred2, average='macro')
f1scr2 = mtr.f1_score(dfy_ts, y_pred2, average='macro')
accr2 = mtr.accuracy_score(dfy_ts, y_pred2)
print('ROC-AUC (by scoring) : ',rocauc2)
print('Precision Macro : ',prec2)
print('Recall Macro : ',recl2)
print('F1 Macro : ',f1scr2)
print('Accuracy : ',accr2)
compare.DecTree = np.array([prec2,recl2,f1scr2,accr2,rocauc2])
# compare
3c) Bagging¶
In [64]:
from sklearn.ensemble import BaggingClassifier
In [276]:
param_grid = {
'n_estimators':[9,11,13,15],
'max_features':[60,70,80,90]
}
grid = GridSearchCV(BaggingClassifier(max_samples=25000),param_grid,refit=True, verbose=True, n_jobs=-1, scoring='recall_macro')
gfit = grid.fit(dfX_tr,dfy_tr)
# print best parameter after tuning
print('Best Params : ',grid.best_params_)
# print classification report
grid_predictions = grid.predict(dfX_ts)
print(mtr.classification_report(dfy_ts, grid_predictions))
print("Best score: %0.3f" % gfit.best_score_)
In [65]:
bgcl = BaggingClassifier(max_features=80, max_samples=25000, n_estimators=13)
bgcl = bgcl.fit(dfX_tr, dfy_tr)
print('Training Score : ',bgcl.score(dfX_tr, dfy_tr))
print('Test Score : ',bgcl.score(dfX_ts, dfy_ts))
y_pred3 = bgcl.predict(dfX_ts)
cmat3 = mtr.confusion_matrix(dfy_ts,y_pred3, labels=[1,0])
dfcm3 = pd.DataFrame(cmat3, index=['1','0'],columns=['1','0'])
plt.figure(figsize=[5,3])
sns.set(font_scale=1.2)
fig = sns.heatmap(dfcm3, annot=True, fmt='d',linewidth=0.5, cbar=False)
plt.tick_params(axis='both', which='major', labelbottom = False, bottom=False, top = False, labeltop=True)
plt.ylabel('Predicted\n')
plt.show()
In [66]:
roc3 = ROCAUC(bgcl)
roc3.fit(dfX_tr, dfy_tr)
roc3.score(dfX_ts, dfy_ts)
Out[66]:
In [67]:
# print classification report
print(mtr.classification_report(dfy_ts,y_pred3))
rocauc3 = roc3.score(dfX_ts, dfy_ts)
prec3 = mtr.precision_score(dfy_ts, y_pred3, average='macro')
recl3 = mtr.recall_score(dfy_ts, y_pred3, average='macro')
f1scr3 = mtr.f1_score(dfy_ts, y_pred3, average='macro')
accr3 = mtr.accuracy_score(dfy_ts, y_pred3)
print('ROC-AUC (by scoring) : ',rocauc3)
print('Precision Macro : ',prec3)
print('Recall Macro : ',recl3)
print('F1 Macro : ',f1scr3)
print('Accuracy : ',accr3)
compare.Bagging = np.array([prec3,recl3,f1scr3,accr3,rocauc3])
# compare
3d) AdaBoost¶
In [68]:
from sklearn.ensemble import AdaBoostClassifier
dtr3 = dtc(criterion='gini',splitter='best',max_depth=3)
dtr4 = dtc(criterion='gini',splitter='best',max_depth=4)
dtr5 = dtc(criterion='gini',splitter='best',max_depth=5)
In [65]:
param_grid = {
'learning_rate':[0.9,1.0,1.1,1.2,1.5],
'n_estimators':[40,50,75]
}
grid = GridSearchCV(AdaBoostClassifier(base_estimator=dtr4,algorithm='SAMME.R'),param_grid,refit=True, verbose=True, n_jobs=-1, scoring='recall_macro')
gfit = grid.fit(dfX_tr,dfy_tr)
# print best parameter after tuning
print('Best Params : ',grid.best_params_)
# print classification report
grid_predictions = grid.predict(dfX_ts)
print(mtr.classification_report(dfy_ts, grid_predictions))
print("Best score: %0.3f" % gfit.best_score_)
In [69]:
abcl = AdaBoostClassifier(base_estimator=dtr4,algorithm='SAMME.R',learning_rate=1.0,n_estimators=50)
abcl = abcl.fit(dfX_tr, dfy_tr)
print('Training Score : ',abcl.score(dfX_tr, dfy_tr))
print('Test Score : ',abcl.score(dfX_ts, dfy_ts))
y_pred4 = abcl.predict(dfX_ts)
cmat4 = mtr.confusion_matrix(dfy_ts,y_pred4, labels=[1,0])
dfcm4 = pd.DataFrame(cmat4, index=['1','0'],columns=['1','0'])
plt.figure(figsize=[5,3])
sns.set(font_scale=1.2)
fig = sns.heatmap(dfcm4, annot=True, fmt='d',linewidth=0.5, cbar=False)
plt.tick_params(axis='both', which='major', labelbottom = False, bottom=False, top = False, labeltop=True)
plt.ylabel('Predicted\n')
plt.show()
In [70]:
roc4 = ROCAUC(abcl)
roc4.fit(dfX_tr, dfy_tr)
roc4.score(dfX_ts, dfy_ts)
Out[70]:
In [71]:
# print classification report
print(mtr.classification_report(dfy_ts,y_pred4))
rocauc4 = roc4.score(dfX_ts, dfy_ts)
prec4 = mtr.precision_score(dfy_ts, y_pred4, average='macro')
recl4 = mtr.recall_score(dfy_ts, y_pred4, average='macro')
f1scr4 = mtr.f1_score(dfy_ts, y_pred4, average='macro')
accr4 = mtr.accuracy_score(dfy_ts, y_pred4)
print('ROC-AUC (by scoring) : ',rocauc4)
print('Precision Macro : ',prec4)
print('Recall Macro : ',recl4)
print('F1 Macro : ',f1scr4)
print('Accuracy : ',accr4)
compare.AdaBoost = np.array([prec4,recl4,f1scr4,accr4,rocauc4])
# compare
3e) Gradient Boosting¶
In [72]:
from sklearn.ensemble import GradientBoostingClassifier
In [ ]:
param_grid = {
# 'loss':['deviance', 'exponential'],
'learning_rate':[0.05,0.1,0.5],
'n_estimators':[40,50,60],
'subsample':[0.1,0.5,1.0],
# 'criterion' : ['friedman_mse', 'mse', 'mae'],
'max_depth':[10,20,50],
'min_impurity_decrease':[0.001,0.1,0.2]
# 'max_features':['auto', 'sqrt', 'log2']
}
grid = GridSearchCV(GradientBoostingClassifier(criterion='friedman_mse',loss='deviance',max_features='auto'),param_grid,refit=True, verbose=True, n_jobs=-1, scoring='recall_macro')
gfit = grid.fit(dfX_tr,dfy_tr)
# print best parameter after tuning
print('Best Params : ',grid.best_params_)
# print classification report
grid_predictions = grid.predict(dfX_ts)
print(mtr.classification_report(dfy_ts, grid_predictions))
print("Best score: %0.3f" % gfit.best_score_)
In [73]:
gbcl = GradientBoostingClassifier(criterion='friedman_mse', learning_rate=0.1, loss='exponential', max_depth=5, max_features='auto', min_impurity_decrease=0.2, n_estimators=60, subsample=0.2)
gbcl = gbcl.fit(dfX_tr, dfy_tr)
print('Training Score : ',gbcl.score(dfX_tr, dfy_tr))
print('Test Score : ',gbcl.score(dfX_ts, dfy_ts))
y_pred5 = gbcl.predict(dfX_ts)
cmat5 = mtr.confusion_matrix(dfy_ts,y_pred5, labels=[1,0])
dfcm5 = pd.DataFrame(cmat5, index=['Yes','No'],columns=['Yes','No'])
plt.figure(figsize=[5,3])
sns.set(font_scale=1.2)
fig = sns.heatmap(dfcm5, annot=True, fmt='d',linewidth=0.5, cbar=False)
plt.tick_params(axis='both', which='major', labelbottom = False, bottom=False, top = False, labeltop=True)
plt.ylabel('Predicted\n')
plt.show()
In [74]:
roc5 = ROCAUC(gbcl)
roc5.fit(dfX_tr, dfy_tr)
roc5.score(dfX_ts, dfy_ts)
Out[74]:
In [75]:
# print classification report
print(mtr.classification_report(dfy_ts,y_pred5))
rocauc5 = roc5.score(dfX_ts, dfy_ts)
prec5 = mtr.precision_score(dfy_ts, y_pred5, average='macro')
recl5 = mtr.recall_score(dfy_ts, y_pred5, average='macro')
f1scr5 = mtr.f1_score(dfy_ts, y_pred5, average='macro')
accr5 = mtr.accuracy_score(dfy_ts, y_pred5)
print('ROC-AUC (by scoring) : ',rocauc5)
print('Precision Macro : ',prec5)
print('Recall Macro : ',recl5)
print('F1 Macro : ',f1scr5)
print('Accuracy : ',accr4)
compare.GradBoost = np.array([prec5,recl5,f1scr5,accr5,rocauc5])
# compare
3f) RandomForest¶
In [76]:
from sklearn.ensemble import RandomForestClassifier
In [327]:
param_grid = {
'n_estimators':[60,70,80,90,100,120,150],
'max_depth':[10,15,20,30,40,50]
}
grid = GridSearchCV(RandomForestClassifier(criterion='gini', class_weight='balanced', max_features='auto', min_impurity_decrease=0.001),param_grid,refit=True, verbose=True, n_jobs=-1, scoring='recall_macro')
gfit = grid.fit(dfX_tr,dfy_tr)
# print best parameter after tuning
print('Best Params : ',grid.best_params_)
# print classification report
grid_predictions = grid.predict(dfX_ts)
print(mtr.classification_report(dfy_ts, grid_predictions))
print("Best score: %0.3f" % gfit.best_score_)
In [77]:
rfcl = RandomForestClassifier(criterion='gini', class_weight='balanced', max_features='auto', min_impurity_decrease=0.001, max_depth=20, n_estimators=80)
rfcl = rfcl.fit(dfX_tr, dfy_tr)
print('Training Score : ',rfcl.score(dfX_tr, dfy_tr))
print('Test Score : ',rfcl.score(dfX_ts, dfy_ts))
y_pred6 = rfcl.predict(dfX_ts)
cmat6 = mtr.confusion_matrix(dfy_ts,y_pred6, labels=[1,0])
dfcm6 = pd.DataFrame(cmat6, index=['Yes','No'],columns=['Yes','No'])
plt.figure(figsize=[5,3])
sns.set(font_scale=1.2)
fig = sns.heatmap(dfcm6, annot=True, fmt='d',linewidth=0.5, cbar=False)
plt.tick_params(axis='both', which='major', labelbottom = False, bottom=False, top = False, labeltop=True)
plt.ylabel('Predicted\n')
plt.show()
In [78]:
roc6 = ROCAUC(rfcl)
roc6.fit(dfX_tr, dfy_tr)
roc6.score(dfX_ts, dfy_ts)
Out[78]:
In [79]:
# print classification report
print(mtr.classification_report(dfy_ts,y_pred6))
rocauc6 = roc6.score(dfX_ts, dfy_ts)
prec6 = mtr.precision_score(dfy_ts, y_pred6, average='macro')
recl6 = mtr.recall_score(dfy_ts, y_pred6, average='macro')
f1scr6 = mtr.f1_score(dfy_ts, y_pred6, average='macro')
accr6 = mtr.accuracy_score(dfy_ts, y_pred6)
print('ROC-AUC (by scoring) : ',rocauc6)
print('Precision Macro : ',prec6)
print('Recall Macro : ',recl6)
print('F1 Macro : ',f1scr6)
print('Accuracy : ',accr6)
compare.RandomForest = np.array([prec6,recl6,f1scr6,accr6,rocauc6])
compare
Out[79]:
In [80]:
cmpr = compare.T
cmpr = compare.reset_index()
cmpr = cmpr.melt(id_vars='index',value_vars=['Logit','DecTree','Bagging','AdaBoost','GradBoost','RandomForest'], var_name='Model',value_name='Value')
# cmpr
fig = px.line(cmpr, x='Model', y='Value', color='index', range_y=[0.6,1.0], width=900, height=600)
fig.data[0].update(mode='markers+lines')
fig.data[1].update(mode='markers+lines')
fig.data[2].update(mode='markers+lines')
fig.data[3].update(mode='markers+lines')
fig.data[4].update(mode='markers+lines')
fig.show()
Summary¶
- Given the objective (per the problem statement) is to "increase their hit ratio" - as I understand it to be increasing the ratio of True Positives among the consumers who are likely to subscribe to term deposit - the focus is on building/tuning a model to yield a higher Recall metric.
- Recall_Macro is assumed to be the right metric in order not to be encumbered with an imbalanced dataset.
- Comparison chart above shows the metrics of the six models built, Random Forest has the highest Recall_Macro score and ROC_AUC score and would be the right choice for predictions in this usecase